NOTE:This program was formerly known as Web Grabber.
by Richard Burgess
NOTE: You can now register Web Retriever through my web page. Go to the about box and click register while you have Netscape open.
GOODIES STILL TO COME
• AppleScript support.
• Enable those darn grab cgi and auto-save options.
• A Finder type list of the site's directory structure.
• Ability to save your settings.
• Built-in networking, no piggy-backing off of netscape.
• Java, java, java ...
• Don't you like to be surprised?
Buggy Stuff, or How I Learned to Stopped Worrying, and Love the Bug
If you have any problem using Web Retriever let me know. If it chokes on a particular site (it always seems to be the one you *really* want:) please point me to it.
Introduction
As you probably quickly realized when you first tried to save a web page, all that is downloaded to your hard drive is the html document itself, not the graphics or other documents that the page is linked to. Web Retriever allows you to copy the complete web page, graphics, links, and all.
Using Web Retriever
Currently, Web Retriever requires NetScape in order to operate—and as such, is slightly hamstrung without direct net access from within the app itself, the next version will probably incorporate some java code to do this. Using Web Retriever is very simple . When you find a web page you want to save, switch to Web Retriever and hit the Fetch! button. The directory structure of the remote site will replicated on your hard drive and an alias to the page you grabbed will be put in the download folder.
Basic Settings
Grab Files on Other servers—Check this option if you want to grab files that are a different server than the page that is pointing to it. Be aware, however, that if you don't have the Make Absolute URL's Relative option selected then the referring page will still point the file's original location on the web and not to the copy on your local disk.
Make Absolute URL's Relative —With this option checked, all absolute url's will be converted to relative url's so that the links will point to correct files when you view the pages off-line. Note: this also effects what is done with the BASE tag, if there is any. If this option is not checked, the BASE tag will not be modified.
Max search layers—This allows you choose how deep you want to go when saving documents that are linked to this web page. Note that non-text files (and javascript files) are considered "on the same level" as the referring page, so, for example, if you specify 0, the images on the page will be downloaded but not links to other pages. Also with frame documents, the html files making up the frame are considered to be on the same level as the frameset page.
Overwrite Existing Files—With this option checked, Web Retriever will over-write files that already exist on your hard drive from previous grabs.
Grab Images—With this option enabled, background images and images in IMG tags are downloaded. In order to download images referenced with an anchor tag, you will need to enable the MIME type filter, see below.
Grab CGI Documents—Web Retriever will attempt to get documents generated by cgi programs, see note below.
Download Folder—This allows you to select a folder in which you want the files saved to. Also, aliases to the pages you grab are put in this folder. Note: the alias will be cryptically named, so it'd probably be a good idea to rename it something meaningful.
MIME Type Filter
Clicking this button brings up the MIME type filter dialog box. Click "Enable MIME type filter" to load up all the MIME types Internet Config knows about. Then double-click on the types you want to download, and/or click the quick settings checkboxes.
Quick Settings - These settings will let you get a general class of files: Applications, Sounds, and Video.
Open Internet Config—This will launch Internet Config, allowing you to set various Internet related options. For our purposes, only the MIME type/extension mappings are important (at present, you'll have to restart WR for them to take effect, tho).
Notes, or They're Not Bugs, They're Features!
• If Netscape fails to download a file for some particular reason, the file will stay in the download list indefinitely. If you see that there are zero files in the queue and the progress dialog stays open, just hit cancel to get rid of it.
• Macintosh filenames are limited to 31 characters, so if a particular file or directory name exceeds that limit then you're SOL. (I suppose I could try to fix this at some point :)
• Files ending in .cgi are programs executing on the server that dynamically generate documents in response to a request from a web browser, so you may not be able to reliably download these types of documents for off-line viewing within the context of the original page. At present, you'll only be able to get text documents when you do a "Fetch".
• Same sort of deal with Java applets and Javascripts. Depending on the set up, all the required files may not be downloaded. Currently doesn't attempt to search for .java and .class files if the extension is not specified, or search for an ARCHIVE or JAR tag.
• And a big thanks to James Neff for the cool about box... pic and icons.
Future Versions
Web Retriever is very much a work in progress, and I plan to add more features. I'm always trying to improve my programs, so if there are any features you like to see added please let me know. Please send bug reports, comments, suggestions, hate mail, etc. to rickb@intellinet.com.
The latest version of Web Retriever, as well as other cool stuff, can be found on my home page at: http://www.intellinet.com/~rickb/
Shareware fee
All I ask is a measly $5 US. Please include your e-mail address, so I can inform you of updates. You can register via a secure credit card transaction on my home page, or you can use snail mail and send a check to:
Richard Burgess
701 Fletcher Place
Lonoke AR 72086
Some of my other feeble attempts at Free/Shareware
